Urgent need: Batch downloader with control over file names

[Andante54]

We use batch downloading on a regular basis, but our existing programs (GetRight and DownThemAll, a Firefox add-on) don't offer much say over the file names.

This can be a problem when the URL is report.txt&id=12345, and the results are report001, report002, report003, etc. Since the download programs don't go down the list in order (they usually jump around), we have no way of identifying the finished products with those critical ID numbers.

We're looking for EITHER:
1) A batch download program that will allow us to include that ID number (or the entire URL) in the resultant file name, or
2) A program that includes a history, showing me which URL resulted in which file.

We would be extremely grateful for any leads! A key project is stopped until we can resolve this.

Thank you!

 

[Ralston18]

Take a look at Powershell.

Google for scripts and then modify those scripts to customize as necessary.

Example:

https/social.technet.microsoft.com/Forums/windowsserver/en-US/e3a3def9-f854-40ae-b809-bbff48eac903/use-powershell-to-download-a-file-through-batch-file?forum=winserverpowershell

Use words such as "Powershell batch download". Vary the words and phrasing to narrow the results.

I believe that Powershell is being overlooked as a administrative tool especially in professional environments. Probably not good for the average home-end user.

Knowing Powershell may become one of those IT "must haves" if it is not already so.

 

[Andante54]

Take a look at Powershell.

Google for scripts and then modify those scripts to customize as necessary.

Example:

https/social.technet.microsoft.com/Forums/windowsserver/en-US/e3a3def9-f854-40ae-b809-bbff48eac903/use-powershell-to-download-a-file-through-batch-file?forum=winserverpowershell

Use words such as "Powershell batch download". Vary the words and phrasing to narrow the results.

I believe that Powershell is being overlooked as a administrative tool especially in professional environments. Probably not good for the average home-end user.

Knowing Powershell may become one of those IT "must haves" if it is not already so.

Thanks, Ralston, I'll take a look at it this afternoon. You believe it can batch download using the entire URL as a resulting file name? (If you have any hints on how to do so, it would save us a bunch of time!) Thanks so much.

 

[Ralston18]

You are welcome.

And I do believe that Powershell is a very viable option.

However (full disclosure) I still consider myself a novice Powershell user. Feel I could cobble some script together but would not trust it to be particularily elegant or efficient. And certainly would not use you (nor anyone for that matter) as a guinea pig.

What I will suggest is that you list out step-by-step what needs to be done.

The flow, if I understand the requirements correctly, would be similar to the following:

Create a constant (string) holding the URL.

Download/copy the file(s) to the target folder.

Rename the file(s) by concatenting the URL with the existing filename.

Then newly named files can be sorted and presented in the desired order.

Probably will require a loop to process each downloaded file. Or the use of "Foreach".

https/msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/foreach-object

Then go online to find Powershell scripts that address each step. You might be lucky and find a script that covers most of it.

In any case, if the script is not a perfect match then make your modifications and test. Get the first part to work and then the second part, and so forth.

Just be sure to keep all data backed up. Really best to create a test environment where, if things go wrong, you simple delete all, copy in the test files again, correct/change your script and try again.

Other example scripts and references:

http/stackoverflow.com/questions/40508062/regex-to-capture-an-url

http/www.powershellatoms.com/basic/download-file-website-powershell/

https/www.howtogeek.com/111859/how-to-batch-rename-files-in-windows-4-ways-to-rename-multiple-files/

The preceding links are more for reference purposes than actual use per se. You will need to make all consistent with your environment and needs.

Get each part to work, get them to work together, get it functional, then all can be tided up and further documented within the script. Use # to remark in and identify the necessary steps within your script. An outline to start with.

As you work on each step things will come together and you will end up with a working, customized script that you know and understand.

And can modify or reuse as circumstances warrant.

 

[Andante54]

You are welcome.

And I do believe that Powershell is a very viable option.

However (full disclosure) I still consider myself a novice Powershell user. Feel I could cobble some script together but would not trust it to be particularily elegant or efficient. And certainly would not use you (nor anyone for that matter) as a guinea pig.

What I will suggest is that you list out step-by-step what needs to be done.

The flow, if I understand the requirements correctly, would be similar to the following:

Create a constant (string) holding the URL.

Download/copy the file(s) to the target folder.

Rename the file(s) by concatenting the URL with the existing filename.

Then newly named files can be sorted and presented in the desired order.

Probably will require a loop to process each downloaded file. Or the use of "Foreach".

https/msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/foreach-object

Then go online to find Powershell scripts that address each step. You might be lucky and find a script that covers most of it.

In any case, if the script is not a perfect match then make your modifications and test. Get the first part to work and then the second part, and so forth.

Just be sure to keep all data backed up. Really best to create a test environment where, if things go wrong, you simple delete all, copy in the test files again, correct/change your script and try again.

Other example scripts and references:

http/stackoverflow.com/questions/40508062/regex-to-capture-an-url

http/www.powershellatoms.com/basic/download-file-website-powershell/

https/www.howtogeek.com/111859/how-to-batch-rename-files-in-windows-4-ways-to-rename-multiple-files/

The preceding links are more for reference purposes than actual use per se. You will need to make all consistent with your environment and needs.

Get each part to work, get them to work together, get it functional, then all can be tided up and further documented within the script. Use # to remark in and identify the necessary steps within your script. An outline to start with.

As you work on each step things will come together and you will end up with a working, customized script that you know and understand.

And can modify or reuse as circumstances warrant.

Omg, that sounds like more work than I'm prepared for at the moment.

Here's all I had wanted.

Files are online in the following format:
www.sitename.com/report.txt&id=12345
www.sitename.com/report.txt&id=48273
www.sitename.com/report.txt&id=01843

Each id is a random five-digit number. I would simply like to download, keeping that number as part of the filename.

OR, find a batch download program with a history, so I can see which new filename belongs to which id number.

Though I do appreciate your time, Ralston, I don't believe I have the energy to learn a new script at this point. Darn it.

But again, you were so kind to write back. Thank you! (If you think of something simpler, please let me know!)

 

[Ralston18]

Agree: may be a lot of work. Like simple whenever possible.

Powershell sometimes can be scarily simple....

Is there a batch file for each URL or does the batch file "walk through" some list of URLs?

Does each listed URL represent just one file? E.g. www.sitename.com/report.txt&id=12345 become report001
and the next UR(...48273) becomes report002 and so forth.

Or are the end results of your batch file on www.sitename.com/report.txt&id=12345 being multiple files being report001, report002, and then the next URL starts report003, report004, report005?

Very likely can parse the paths and filenames for the unique ID numbers and concatentate each with the corresponding filename.

I tinkered with some script and does not take much, for example, to go to one website location, download a file(s), and rename that file in some manner based on the source URL.

One thing at a time.

Invoke-WebRequest can easily be tested. Here is what I came up with as a test.

# Test Script

$url = 'www.sitename.com/report.txt&id=12345'

$output ='C:\Downloads\TempReports'

Invoke-WebRequest -Uri $url -OutFile $output

#End test script

Can you safely edit and run that script and provide me with any errors or the files that end up in C:\Downloads\TempReports? You may need to create the destination folder TempReports. [Note: I do not need to know nor want to know the real sitename. Hopefully that is the only edit on your end]

If you are unable to do so or do not feel comfortable doing so then do not do anything - will take no offense on this end.

Anyway if the script works as I hope, then I will see if I can parse out and concantenate in the random five-digit number.

 

[Andante54]

Thank you, Ralston -- I apologize I was gone for a few days.

I am happy to try your test! And yes, as I mentioned in my first post, it's one URL per file. Only the five-digit number changes.

Please tell me where I can download Powershell and I'll try your test.

 

[Ralston18]

No problem - those things happen....

You may already have Powershell. Check your computer(s).

If not, since I do not know your environment, best to go to Microsoft's website and download the applicable version.

Admin rights will probably be needed so use a test environment to begin with.

https/www.microsoft.com/en-us/search/result.aspx?q=Powershell&form=MSHOME

This link may be helpful as well:

https/www.iis.net/learn/manage/powershell/installing-the-iis-powershell-snap-in

 

[Andante54]

I'm sorry, Ralston, but those links are all too overwhelming. I'm running Windows XP, SP3 (please don't laugh), and nowhere does it say on those pages which version of Windows you need. I then googled "PowerShell for Windows XP" trying to find something more specific, but it became too much.

Truth be told I'm not feeling great these days, and was hoping for a solution without undue brainpower. But I'm afraid that learning a new scripting language is over my head at the moment.

(Am I wrong? Would this truly be a cinch to do?)

 

[Ralston18]

No, will not laugh because many folks just have to deal with what they have through no choice of their own.

Also, understand "overwhelmed".

You would need to download Powershell 1.0 or perhaps 2.0 for XP SP3 use.

Reference:

https/support.microsoft.com/en-us/help/968929/windows-management-framework-windows-powershell-2.0,-winrm-2.0,-and-bits-4.0

Go about 1/3 down the page to "System Requirements".

"Cinch" is relative. There may be someone else following this thread who could provide us both with one of those "Eureka" moments. Something I missed perhaps. Fine with me - I will learn as well.

Automating a manual process can be a cumbersome process when you get down to the step by step details.

I have had it go both ways: something like the little test script may work perfectly or with very little additional tweaking. Other times: midnight oil.... And after that, someone comes along and and shows me another way (usually simpler).

Would suggest it is a matter of priorities now. Temporarily set it aside until your circumstances improve, continue to search for other options, delegate the assignment....

 

[Andante54]

Thank you for the kind message, Ralston. I appreciate your understanding.

I think at this point the only way I could use PowerShell is if someone gave me the exact script to use. I'm afraid it's simply beyond me to do it myself right now.....

On another thread, I was directed to both wget and curl. (Do you know them?) Someone there helped me find the correct options to use wget, but I was curious to see if PowerShell might be easier. Perhaps I should return to the wget solution....it's a bit of a long way but at least it works!

Don't know anything about curl, or whether it might be better.

Finally, your advice is well taken. There actually is no hurry on this, and I could indeed do it later. Thank you, my mysterious friend. ;-)

 

[Ralston18]

Again you are welcome.

Unfortunately do not know them. And do not know if they are "easier" per se. Always trade-offs...

Have always believed that being able to code my own "tools" provides more flexibility and greater control. Sometimes true - sometimes not, and sometimes there is a third party tool already out there.

Overall, I find Powershell to be a very demanding tool but quite powerful. And its capabilities seem to be growing. A few googles addressing various problems has shown me that there is still much therein for me to learn....

Had an afterthought: Go online and contact GetRight and DownThemAll.

They may have some tool or other recommendation that may address your automation requirements. No harm in asking.

Likewise check their forums and FAQs.

 

[Andante54]

GetRight is pretty inflexible, but DTA has a lot of flexibility! I sent them a detailed email a few weeks ago, and am still haven't heard back.

Couldn't find any other mention online of this issue, though I'm not familiar with the appropriate forums.

I completely and totally agree with you that knowing a language to program gives you many many levels more power in manipulating your environment. That's why I learned the Excel macro language back in the day (and still use it!). It's amazing what I've been able to accomplish with it (including a magic trick that blew away some magician friends).

But at the moment, I'm not up to learning a new one. It sure would be fun someday to do so, however. ;-)

 

[ronald223]

I would suggest to use "KrojamSoft BatchRenameFiles" toolfor bulk renaming