Advice for how to search/replace a 2gb text file - Webmaster Forums - Webmaster forum for HTML, PHP, ASP, CSS and more

derricks2 · 07-02-2011, 09:21 PM

I am trying to import a data feed text file into a database. The text file uses " within it. I need to replace these with ". The problem is that the text file is almost 2gb and has 1.25 million lines.

Obviously normal text editors aren't going to get this job done and I have been searching for near an hour now for a program that will accomplish this, but none of the ones I have found thus far have been able to deliver what they claim to be able to accomplish.

I am sure that with data feeds being more of a common thing these days, that someone here would know how to get this done so I can import this into the database.

Thanks in advance.

Rukbat · 09-16-2011, 04:26 PM

It depends on the format of the data file, but basically you write a program to read the file line by line and do the replacements on each line. Use whatever programming language you're comfortable with (there's probably no language that can access the database that won't work), or whatever one the programmer you have doing the job is comfortable with.

We do this with XML files, xls files, text files, CSV files (and what some people call CSV files that I call plain old corruption). If a programming language can read the file, replacing a character in it during the database load is trivial. (I've only done it in C, Pascal, VB and PHP, because those were the languages I was working in when I needed to do it, but any language will work.)