NFT - Anybody here provide (or know of ) data scrubbing services? Referral $$$ paid

[Trivers]

Would prefer to spend $$$ with fellow CPers.

We have 43 databases averaging 50K each that need to be cleaned up.

Seeking consultant.

Referral $$$ paid to your paypal account

Thanks

[58-4ever]

Let me know what kind of services you need. Are you in the KC Metro?

[stumppy]

I got a gal that comes by when I call and cleans my pipes. She's pretty talented, I'm sure she could clean your databases too.......while wearing a french maid outfit.

[58-4ever]

Shoot me a PM

[stonedstooge]

Can we keep your porn?

[Rain Man]

Cleaned up how?

[CaliforniaChief]

http://www.troll.me/images/jesse-pinkman/magnets-bitch.jpg

[TLO]

This thread had potential.... but alas.

[srvy]

Better call Saul.

[srvy]

If its really bad The Wolf.

<iframe width="560" height="315" src="//www.youtube.com/embed/IgzFPOMjiC8" frameborder="0" allowfullscreen></iframe>

[AustinChief]

Gonna need more details. Are you looking at something that could be automated or something that requires a manual once-over?

[Trivers]

Thank you for your responses.

Here are the details:

We are sending emails to list of insurance agents in all fifty states.

The databases are public records.

The first list we need cleaned up is from WisCONsin. (How the natives pronounce it.)

http://oci.wi.gov/agentlic/agntlist.shtml

Scroll down to bottom of page:

Format 2 - Agents by Company Appointments

There are 17 files. First one is al_1st-al.exe. We figured out how to separate columns, and remove the records without email addresses.

1) We need all the first and last name duplicates removed; and 2) words turned into lower cases.

We would prefer a way to automate this whole process.

If interested, please PM.

Thanks

[unlurking]

*1) uniq file > newfile
2) tr '[:upper:]' '[:lower:]' < file > newfile

*(Are you sure you want to remove dupes by name and not just the entire line? If there are two John Smith entries with different info you will lose one. Might be better to remove complete dupe lines instead?)



</pre>

[Trivers]

*1) uniq file > newfile
2) tr '[:upper:]' '[:lower:]' < file > newfile

*(Are you sure you want to remove dupes by name and not just the entire line? If there are two John Smith entries with different info you will lose one. Might be better to remove complete dupe lines instead?)



</pre>


Good catch! Yes.

Thank you!

[unlurking]

If you have a bash shell, this will work...


cat infile | cut -c101-135,136-160,350-420 | tr '[:upper:]' '[:lower:]' | sed 's/ \s\+/,/g' > outfile
uniq infile outfile
EDIT: This was tested using the "al_1st-al.exe" file from the site you linked to. Only 8 dupe lines are dropped.

[ChiefRocka]

fdisk

[Simply Red]

Speak with Stryker - & I'd like to apologize in advance if you've already found a solution.

But Stryker will (at a minimum) be able to guide you to someone. I just don't recall for sure if he's still in KC Metro.

Good luck to you!

[Dave Lane]

http://www.midlandhardware.com/thumbnail.asp?file=assets/images/products/730393.jpg&maxx=350&maxy=350

This should scrub it for you. PM me for paypal fee,



:)

[Trivers]

*1) uniq file > newfile
2) tr '[:upper:]' '[:lower:]' < file > newfile

*(Are you sure you want to remove dupes by name and not just the entire line? If there are two John Smith entries with different info you will lose one. Might be better to remove complete dupe lines instead?)



</pre>

Check your PM. Trying to reach you.

[ptlyon]

Magnet

[Window Licking Whiner]

Couldn't you just use awk? Instead of cut? awk '{print $(field#)}' | then do the translation from upper to lower