FoxBin2PRG handle records in DBFs

Topics: General
Coordinator
Jun 2, 2014 at 4:11 PM
I have customized my copy of FoxBin2PRG to not just output the structures of DBFs but the records as well. It doesn't currently handle merging, just output. If anyone else has any interest in this, I can post my version.
Coordinator
Jun 2, 2014 at 7:57 PM
Doug --

Yes, I would be interested in having FoxBin2PRG not just output the structures of DBFs but the records as well.



--
Jim Nelson

(805) 498-9195 (preferred)
(720) 837-3536 (cell)
Developer
Jun 15, 2014 at 12:06 PM
Hi Doug:

I never looked at the "Discussions" tab up to now :-)

It would be interesting to have your modified version!


Thanks!
Coordinator
Jun 15, 2014 at 12:55 PM
I uploaded it as an "experimental" version. To see the changes I made, look for "*** DH" comments. As I mentioned, it only outputs records, not merges them back in.
Developer
Jun 15, 2014 at 1:20 PM
Hi Doug:

I don't know where to find this upload. I've looked at "Source Code" tab and "Issues", but didn't find it.
Coordinator
Jun 15, 2014 at 4:23 PM
It's on the FoxBin2PRG releases page.
Developer
Jun 15, 2014 at 8:24 PM
Hi Doug:

I've looked at your changes, and they look good! The only reason I haven't included this functionality from the beginning is that the main purpose of FoxBin2Prg is to be used as diff and merge tool, so the data is generally not used in this scenario.

But you give an idea that can serve for this purpose: I've added a new "run_AfterCreate_DB2" property that fires the indicated VFP program or procedure each time a DBF is converted to DB2, so when using FoxBin2Prg as object, this procedure receives the name of the cursor (which is open at this time) and the datasessionid, so anyone can make his own routine for exporting data. In your example you have used a manual construction of XML, other can use CURSORTOXML or even other implementations.

On version v1.19.19 I've added the opposite "run_AfterCreateTable" property, which indicated program or procedure is fired each time a DB2 is converted to DBF. In this case I know a developer that is using it for a personal backup system, to load the DBFs from external files. There is an example demo program on TESTS folder called "demo_hook_dbf.prg" showing how to use this functionality.

I think that having this possibility of calling an external program permits broader use cases that can be applied just when neccesary and not force to always do it.

Is this useful to your use case?
Developer
Jun 15, 2014 at 9:28 PM
I've updated the documentation with this topic and an example, at the end:

FoxBin2Prg Internals and Configuration

Best regards!
Coordinator
Jun 16, 2014 at 1:39 PM
The reason I added this capability is because I am interested in knowing what changes were made in configuration data stored in DBF files from one version to another rather than in merging those changes. However, you changes should make the merging easier if I ever need to do that.
Developer
Jun 16, 2014 at 2:15 PM
Doug, the reason I suggest you implement this externally, is that in one side you don't want this for all the tables, just for that who have configuration data, which you can filter on an external program, and on the other side, you miss the bug fixes on next versions.

May be I'm missing something, but I think that an external program can do the same with the adventadge that on next versions you can keep updating without loosing this functionallity.
Coordinator
Jun 16, 2014 at 2:39 PM
I had missed the capability of doing it in an external program. I'll give that a try. I've removed my experimental version from the downloads page. Thanks.
Jul 11, 2014 at 4:29 AM
Hi guys .. umm .. i'm a little confused .. how do I make bin2prg to convert the data in dbf files too ?
It doesn't convert by default using the exe and the vbs script from SendTo .. do I have to change something in the "foxbin2prg.cfg" file?
Developer
Jul 11, 2014 at 7:24 AM
Hi Edy:

FoxBin2Prg does not convert data, just the structures. You have the option to instance it as object and use the events mentioned in this thread to call your personal export/import routine.
The reason is that if you want to compare data, it's more efficient and useful to use your own order to see the differences (each table can have a different order)
When working with Source Code Managers (the main use of FoxBin2Prg) you normally don't checkin tables with data, except some little ones.

Please, read this to know what FoxBin2Prg is:
(https://vfpx.codeplex.com/wikipage?title=FoxBin2Prg&referringTitle=Home)

And read the section "FoxBin2Prg Internals and Configuration" to know how you can use those events to do what you need.

Best regards!
Jul 14, 2014 at 4:25 AM
Ok .. I needed something to convert all the tables in a folder to "comparable" outputs, that i can compare with a tool like Beyond Compare and i had the impression that FoxBin2Prg handles that at least at some basic level, converting a binary dbf into a text file fully ..

Anyway thank you for your response and .. I'll look into your info and/or build a converter for the next time I'll need it .. But from what I understand I can't use that with the SendTo scripts and the exe right ? I have run the code in vfp that creates an object then assign a prg that makes the actual converting to the object's property "run_AfterCreate_DB2" ... that's a little too complicated ..

I do wish you'd add a generic feature to include data (non ordered) in the converting process, maybe one that could be activated with a switch in the configuration file, since I will probably use that to compare two versions of the same databases or the same dbfs ..
Developer
Jul 14, 2014 at 6:49 AM
Hi Edy:

Given that you are not the first looking to export data for comparing with some tool (Doug have proposed a modification in this way), I think that I can add this generic functionallity with a switch on the CFG, so that people that just want to export the data and don't need the table structure can do that using the SendTo scripts.

I take a look at this and make you know about it.


Best regards!




2014-07-14 6:25 GMT+02:00 edyshor <[email removed]>:

From: edyshor

Ok .. I needed something to convert all the tables in a folder to "comparable" outputs, that i can compare with a tool like Beyond Compare and i had the impression that FoxBin2Prg handles that at least at some basic level, converting a binary dbf into a text file fully ..

Anyway thank you for your response and .. I'll look into your info and/or build a converter for the next time I'll need it .. But from what I understand I can't use that with the SendTo scripts and the exe right ? I have run the code in vfp that creates an object then assign a prg that makes the actual converting to the object's property "run_AfterCreate_DB2" ... that's a little too complicated ..

I do wish you'd add a generic feature to include data (non ordered) in the converting process, maybe one that could be activated with a switch in the configuration file, since I will probably use that to compare two versions of the same databases or the same dbfs ..

Read the full discussion online.

To add a post to this discussion, reply to this email ([email removed])

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com


Developer
Jul 14, 2014 at 7:03 AM
Hi Doug:

Now I'm taking a second look at this, I think that you (and Jim) were right about this export functionallity inside, and that, taking adventage of multi-configuration feature, can be activated or deactivated to do this in any folder, or just in one folder.

I didn't realize that unordered data can be useful to some people.

Best regards!
Developer
Jul 15, 2014 at 8:49 PM
Edited Jul 16, 2014 at 11:56 AM
Hi Edy:

I've made the modifications to support this functionallity (based mostly on Doug Hennig code).

How it works:
On the dbfs directory create a foxbin2prg.cfg file with this inside:
DBF_Conversion_Support: 4

And that's it, send your DBF to FoxBin2Prg and the generated db2 have the data inside.

Extra feature:
If you create a file with your DBF name plus ".cfg", then you can customize the order in which records are generated.

Example:
You have CUSTOMER.DBF and then create CUSTOMER.DBF.CFG with the index key inside, as this:
Custno

Then the records in the db2 file are ordered by Custno field. You can use any valid expression, but order is allways ASCENDING.

Download the preview from this link:
FoxBin2Prg v1.19.27 Preview (DBF Data Export)


If anyone can confirm that this is ok, I will release it soon.


Note:
Thanks Doug for your code, it was very helpful.
Jul 21, 2014 at 1:28 AM
Edited Jul 21, 2014 at 1:29 AM
fdbozzo .. thanks for deciding to include this feature in FoxBin2Prg but .. let me explain the way i use it for comparison:

I use a program that helps me compare two files side by side .. since it can't natively compare two dbf files (or sfx, frx, vcx for that mater) i have to convert those and compare the generated text files. I do this very often and that's why I use the send to scripts of FoxBin2Prg. Also I don't use it only to compare two versions of my data folder in my development version of the project.

That being said, the need to create configuration file for a certain dbf or in the folder the dbf is in its .. cumbersome and in my opinion over the top.
If that option could also be turned on in the (main) config file that is located where the foxbin2prg.exe is (the one used in the SendTo scripts) that would be great for me.

You could allow the user to add the option in the main config file and then foxbin2prg will do the full convert everywhere or if the option is missing the program will do the full convert under the conditions you described above (selectively ?!).



In the situation where the config file is in the folder where the dbfs are - lets say my project data folder - you could allow the user to list what dbfs need to be converted with data (or excluded) and even go further to allow the user to specify a filter condition for the data to be exported in the "dbf_name.dbf.cfg" file - something like "is_initial = .T." and only the records that validate under that condition will be exported.

Here's a use case: I have 10 dbf files, where 3 have a fixed set of options (values for selction comboboxes) that i want to be exported in full and 2 dbf also have values to populate combos or lists but 5 are predefined and the user can add more to them but he won't touch those 5 predefined values. In those 2 dbfs, the 5 rows that are predefined will have .T. in the "is_initial" field. In these two dbfs I only want those predefined rows to be exported.

So, the file "foxbin2prg.cfg" in my data directory will have these options:
DBF_Conversion_Support: 4
DBF_Conversion_Included (suggestion): cities.dbf, streets.dbf, parks.dbf, products.dbf, titles.dbf (3 fixed and 2 that could be expanded)

I'd like to have the option to add "DBF_Conversion_Included" or "DBF_Conversion_Excluded"

Then i'll also have two more files: "products.dbf.cfg" and "titles.dbf.cfg" both having thse options:
Conversion_Order: cust_order
Conversion_Condition: is_initial = .T.

Hope you'll consider some of it at least, otherwise i don't really see the need to add a special config file to may data folder and even create more for my dbfs unless that would help me to fine tune the conversion process ..

Thanks.
Developer
Jul 21, 2014 at 5:35 AM

Hi Edy:

I could add this functionality, but now I could get a regression problem because you didn't give me this feedback last week when I notified you that a preview was available, and before I released the final version.

So I will take your suggestions into consideration but, please, subscribe to this thread so you get notified instantly and this way you can give me the feedback I need faster, in the development stage, before people could be affected by a final version.

If you could do this, then I can think about it seriously.

Best Regards,

Fernando D. Bozzo

Coordinator
Jul 21, 2014 at 2:09 PM
Fernando --

I have not been following this discussion very closely until now but would like to add my two cents in.

First, I think that it is unreasonable for you to expect that everybody affected by your project will always be following your thread and able to comment on improvements you are suggesting. Apparently, Edy and I missed a brief comment period about a desirable enhancement.

Second, I can't imagine that there can be much of a regression problem for users of a feature just announced last week.

And last, I would have to agree with Edy's suggestion, as I certainly would expect that FoxBin2PRG would have a single global setting (with possible local overrides) to use this feature.

Jim


Developer
Jul 21, 2014 at 2:31 PM
Edited Jul 21, 2014 at 4:29 PM
Hi Jim: I answer on your post.

El 21/07/2014 16:09, "JimRNelson" <[email removed]> escribió:
From: JimRNelson

Fernando --

I have not been following this discussion very closely until now but would like to add my two cents in.

First, I think that it is unreasonable for you to expect that everybody affected by your project will always be following your thread and able to comment on improvements you are suggesting. Apparently, Edy and I missed a brief comment period about a desirable enhancement.
Jim, you are wrong on this, this is not an improvement "I am suggesting", it is an improvement proposed by Edy, so the minimum I expect, is that the requester give some feedback on what is asking for. I'm not asking everyone for this to follow up.
Second, I can't imagine that there can be much of a regression problem for users of a feature just announced last week.
You are right on this, but the more time passes, more probability of regression.
And last, I would have to agree with Edy's suggestion, as I certainly would expect that FoxBin2PRG would have a single global setting (with possible local overrides) to use this feature.
You are not following my releases very close, didn't you? :-D

This is something currently available, and enhanced by the multi-configuration feature.

The main CFG file always was modifiable, and can be partially or full overriden by directory (I've documented this on VFPX help pages)

The only warn is that this modification should take into account that this file is distributed too, so should be backed up.

Regards. -
Jul 21, 2014 at 3:41 PM
I'm sorry man .. I've been real busy and when that happens I tend to cut myself from the world for longer durations of time .. Jim can attest that I can't always be trusted to reply as fast as its desirable regarding a preview / alpha version. But I will help when I can.

For my needs the global setting is a must. The other suggestions are optional but would be great to have - and since you already have part of the structure in place adding those might be easier.
Developer
Jul 21, 2014 at 3:49 PM
Ok Edy, don't worry:

Global CFG is what you have by default, so can use it right now.

The other stuff, I will look at it soon.


Best regards!




2014-07-21 17:42 GMT+02:00 edyshor <[email removed]>:

From: edyshor

I'm sorry man .. I've been real busy and when that happens I tend to cut myself from the world for longer durations of time .. Jim can attest that I can't always be trusted to reply as fast as its desirable regarding a preview / alpha version. But I will help when I can.

For my needs the global setting is a must. The other suggestions are optional but would be great to have - and since you already have part of the structure in place adding those might be easier.

Read the full discussion online.

To add a post to this discussion, reply to this email ([email removed])

To start a new discussion for this project, email [email removed]

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com


Developer
Jul 21, 2014 at 9:54 PM
Edited Jul 21, 2014 at 10:27 PM
Hi Edy:

This is the preview version with what you asked for:

FoxBin2prg.cfg (Main CFG or Directory CFG)
New foxbin2prg.cfg options for filtering tables from conversion using one or multiple conbinations of filemasks:

DBF_Conversion_Included: <filemask>[ ,<filemask> [ , ... ] ]
DBF_Conversion_Excluded: <filemask>[ ,<filemask> [ , ... ] ]

Example of multiple file masks (separte with ","):
PET*.*, ??ME.DBF, ???.DBF, ?.*
.

New filename.dbf.cfg configuration options
This options can be used in any combination inside a dbf particular cfg file:

DBF_Conversion_Order: cust_order
DBF_Conversion_Condition: is_initial = .T


Note 1: "Condition" can be any valid VFP condition
Note 2: Options that begin with asterisk * will be ignored, so * this can be used to comment a setting without deleting it.

Download Preview at:
FoxBin2Prg v1.19.28 PREVIEW (DBF data filters)

When you test it, tell me if it's ok,

Best regards,

Fernando D. Bozzo
Developer
Jul 24, 2014 at 12:18 PM
Hi Edy:

Have you seen this? Is working as expected?

Regards,

Fernando D. Bozzo
Developer
Jul 26, 2014 at 5:21 PM
Just released in v1.19.28

Bye.-
Nov 19, 2014 at 1:53 AM
Edited Nov 20, 2014 at 4:07 AM
Hi Fernando,

I have been playing around with foxbin2prg trying to find the differences of a database (dbc + many dbfs) at two different versions (like what changes occurred in the last couple of days) and i found a couple of issues.

First there is an error and i submitted an issue for that.. after posting it I tried converting that big dbf (11 Mb) with "foxbin2prg.prg" in my vfp ide and it seems the error occurs when the code tries to add a string to an already full string. My only idea for an workaround would be to add all strings into a temp file with StrToFile( temp_file ,1 ) - using the additive option - then move that temp file to its destination. Since involving the disk would introduce a speed penalty a check for the file size should be made to ensure that this method is really needed.

Then the option "DBF_Conversion_Excluded" doesn't seem to work, I have placed a cfg file in the database folder and set DBF_Conversion_Excluded: mydbfname.dbf but foxbin2prg still processes the file and throws the error.

Also in every .db2 file you create a tag "<LastUpdate>" which is useful but not when trying to find if and what files are different - all the db2 files are shown as different when most of them have the difference only in the values of <LastUpdate>. Is there an undocumented setting that i can add to foxbin2prg.cfg so it doesn't create that tag anymore for converted dbfs ? If not, could you add one please ?

Best regards,
Eduard.

Update: I did some experiments with saving a string to file once it gets to a certain length and found that a string variable can contain a maximum of about 27 856 895 (maybe a little more) and if the string is saved to file ( with additive parameter for StrToFile() ) once it gets to a size of 10 000 000 chars the process is very slow, if saved once it gets to 5 000 000 is better but still slow and I was satisfied with the speed I got by saving the string to file once it gets to 1 000 000 characters.

I hope it helps,
Eduard.
Developer
Nov 20, 2014 at 8:11 AM
Hi Edyshor:

I've reproduced and found the problem and I'm working on the solution of this.
I've tested replacing the string manipulation (string = string + newString) with direct file updating with FCREATE/FWRITE.... and I'm amazed by the results, so I'm implementing this permanent fix when DBF_Conversion_Support is 4.

Thanks for your suggestions, testing and reporting, I'll be back soon.
Developer
Nov 20, 2014 at 8:52 PM
Hi Eduard:

I've uploaded the Beta5 with your suggestions and the "string too long" correction.

Modifications included:
  • Fixed "string too long" when generating db2 files with DBF_Conversion_Support:4, and enhanced the performance
  • New CFG setting "ClearDBFLastUpdate" (enabled by default) => Values: 0,1
Can't reproduce the "DBF_Conversion_Excluded" problem. I've tested again and works as expected. May be I should be more clear on how it works:
  • Main CFG (that is in FoxBin2Prg.exe/prg dir) controls all settings an all dirs
  • Directory CFG settings affects just the files on the directory on which the CFG resides
If something is not working ok for you, please supply some example with your setup and directory config, to reproduce locally.

Tell me how it goes,

Regards.-
Nov 21, 2014 at 11:04 AM
Hi Fernando,

I confirm that the problem is gone, foxBin2Prg correctly processes the 11Mb dbf file and it doesn't create the <LastUpdate> tag anymore which is good.
As for the problem with exclusions, here is the line in foxBin2Prg.cfg file :

DBF_Conversion_Excluded: tablename1.*, tablename2.dbf && If DBF_Conversion_Support:4, you can specify mnultiple filemasks ...

After running foxBin2Prg I discover that it doesn't create a db2 file for "tablename1.dbf" (I have "tablename1.dbf" and "tablename1.cdx" in the folder) but it still creates a db2 file for "tablename2.dbf" (this one doesn't have a cdx counterpart).
I checked again for names and they are written correctly. Maybe the engine expects a true mask ( with * and/or ? characters ) and since "tablename2.dbf" doesn't contain any it doesn't exclude it ? There is no issue with the fact that I left the comment on the same line, right ?

Regards,
Eduard
Developer
Nov 21, 2014 at 3:49 PM
Hi Eduard:

Using comments && on foxbin2prg.cfg configuration file was not supported until now, but I've taken it as a suggestion and I've implemented it in Beta6

Now you can use comments && in any item of foxbin2prg.cfg configuration file.

Tell me if it's working now with your config,

Regards.-
Developer
Dec 5, 2014 at 10:53 AM
Comments in settings working ok, and released in v1.19.37

Regards.-