How to wget large files from one server to another

TupambAdminOrg [2024.03]

4 Tage her • •

TupambAdminOrg [2024.03]
4 Tage her • •

How to wget large files from one server to another

#fediAdmins
@Friendica Support @Friendica Developers

Hi there,
a technical question not directly related to friendica itself.

Trying to transfer an instance from on (shared hosting) server to a debian/ubuntu VPS that's building up.
The problem now is that the DB is around 3.7 gig so I tried to pull the file with wget.

The server responds with:
rootName@VPShosting:/var/storage# wget https://domain.com/folder/DB.sql.bz2
..
Connecting to domain.com (domain.com)|xxx.xxx.xxx.xxx|:443... connected.
HTTP request sent, awaiting response... 413 Request Entity Too Large
ERROR 413: Request Entity Too Large

Can't find a solution on the web.
Google even manages to point to howTo's behind paywall's or subscribe pages, so much about enshittification ..

Did find nginx references but do not have nginx.

Also found for php.ini the following, obviously the values a relative:

max_input_time = 24000
 max_execution_time = 24000
 upload_max_filesize = 12000M
 post_max_size = 24000M
 memory_limit = 12000M

As of now changed:

 upload_max_filesize = 8000M
 post_max_size = 8000m

and memory limit shows
memory_limit = -1
so that's probably not the point.

The error message stays the same.

The input_time and execution_time should kick in in any case while downloading the file.

Ideas how to solve this?

#fediadmins @Friendica Support @Friendica Developers

Als Antwort auf TupambAdminOrg [2024.03]

utopiArte

Als Antwort auf TupambAdminOrg [2024.03] • 4 Tage her • •

update:
found the following hints pointing out that php-fpm .ini could be involved:
askubuntu.com/questions/332022…

Changes in that .ini and restarting apache2 and the respective -fpm didn't work out either.

Of course we are talking about management with the console over SSH.

Als Antwort auf utopiArte

utopiArte

Als Antwort auf utopiArte • 4 Tage her • •

To make sure that the settings that rule are the ones I changed, I searched for a comand line command to print the respective values within the console:
php -r "echo ini_get('my_value');"
(the print is a littel bit strange tho, just a number in line before the user handle, like:
8000MrootName@VPShosting:~# )

php -r "echo ini_get('upload_max_filesize');";"
8000M
php -r "echo ini_get('post_max_size');"e');"
8000M
php -r "echo ini_get('max_execution_time');"e');"
0
php -r "echo ini_get('memory_limit');"t');"
-1
php -r "echo ini_get('max_input_time');";"
-1

Als Antwort auf utopiArte

utopiArte

Als Antwort auf utopiArte • 4 Tage her • •

Just managed to pull two files of 1.9gig and 1.6gig with the existing setting and the command as displayed, so the limitiation is above 1.9gig and most likely a wget problem itself (?).

Als Antwort auf TupambAdminOrg [2024.03]

Daniel

Als Antwort auf TupambAdminOrg [2024.03] • 4 Tage her • •

If you have SSH access to both systems you can just use SFTP. Wget sounds like the wrong tool for this.

Als Antwort auf Daniel

utopiArte

Als Antwort auf Daniel • 4 Tage her • •

@Daniel

If you have SSH access to both systems you can just use SFTP

Not really.
I do have ftp access to the shared hoster using fileZilla not SSH, so I can use those credentials and SSH with putty over console on the VPS. I was just starting to look into such an option but not sure for the correct command to do so. I'd probably also need to install some ftp package I guess. It's a ubuntu 2022.04 standard setup VPS.

@Daniel

Als Antwort auf utopiArte

friendica (DFRN) - Link zum Originalbeitrag

Rainer "friendica" Sokoll

Als Antwort auf utopiArte • 4 Tage her •

rsync is the answer.

teilten dies erneut

Als Antwort auf Rainer "friendica" Sokoll

utopiArte

Als Antwort auf Rainer "friendica" Sokoll • 4 Tage her • •

@Rainer "friendica" Sokoll

Read that a lot, but I guess that my access to the shared hosting only with a tool like filezilla isn't enough to use resync.

Also, I'm in the middle of nowhere with internet access thru a cellphone located at a specific 20x10cm spot in the roof where I catch a littel bit of the provider antena that's at a distance of like 12km. So downloading 3.7gig to than upload it with scp is kinda impossible.

@Daniel

@Rainer "friendica" Sokoll @Daniel

teilten dies erneut

Als Antwort auf utopiArte

Rainer "friendica" Sokoll

Als Antwort auf utopiArte • 4 Tage her •

You said you have ssh access. That should be enough for god's own rsync 😀
And: rsync can resume (-P option), so you would not have to start over after an interrupted transfer.

teilten dies erneut

Als Antwort auf Rainer "friendica" Sokoll

utopiArte

Als Antwort auf Rainer "friendica" Sokoll • 4 Tage her • •

Not sure if I understand what you are saying @Rainer "friendica" Sokoll.

As for what I saw, on a brief look in conversations on the matter on the web, was that rsync has to be installed on both sides, on both servers. I ony can install it on the VPS, not on the shared hosting where the files is.

@Daniel

@Rainer "friendica" Sokoll @Daniel

teilten dies erneut

Als Antwort auf utopiArte

Rainer "friendica" Sokoll

Als Antwort auf utopiArte • 4 Tage her •

There is a good chance that rsync is already installed on the server side. You may try it.

teilten dies erneut

Als Antwort auf Rainer "friendica" Sokoll

utopiArte

Als Antwort auf Rainer "friendica" Sokoll • 3 Tage her • •

@Rainer "friendica" Sokoll @Daniel The web say's this is not possible:

It is not possible to use rsync on http URLs.
To use the rsync algorithm the client-side rsync needs to interact with a server-side rsync process. This is done either directly, through ssh or (less common) rsh. HTTP is not an option.
stackoverflow.com/questions/33…
Unless I can use sFTP credentials to contact the shared hosting with rsync this doesn't look like an option.

@Rainer "friendica" Sokoll @Daniel

teilten dies erneut

Als Antwort auf utopiArte

Rainer "friendica" Sokoll

Als Antwort auf utopiArte • 3 Tage her •

You wrote:

I do have ftp access to the shared hoster using fileZilla not SSH, so I can use those credentials and SSH with putty over console on the VPS.

So I assumed you have ssh access. If you have and rsync is available an both sides, then you really should give rsync a try.

teilten dies erneut

Als Antwort auf Rainer "friendica" Sokoll

utopiArte

Als Antwort auf Rainer "friendica" Sokoll • 3 Tage her • •

@Rainer "friendica" Sokoll @Daniel

I assumed you have ssh access

Nope.
(actually I detailed that in another answer, I guess you didn't get that displayed)

Right now I most likely will try to get that instance running and up to date on the shared hosting, than try to split up the file storage and than move it in parts.
Another option might be looking for a day with a good cellphone reception and do all of it with scp from over here.
Also, investigating how to do this with sFTP over SSH from the VPS is most likely an appropriate option that add's experience and tools to the general setup, as the wget investigation is kinda "ausgereizt" already.

Last but not least there could be the option to ask anyone else who does have (good fiber optics) connections to get the file onto his VPS and than create a resync setup.
👍

@Rainer "friendica" Sokoll @Daniel

teilten dies erneut

Als Antwort auf utopiArte

Rainer "friendica" Sokoll

Als Antwort auf utopiArte • 3 Tage her •

I assumed you have ssh access
Nope.

[…]

investigating how to do this with sFTP over SSH

I am confused. If you do not have ssh access, sftp will fail.

teilten dies erneut

Als Antwort auf Rainer "friendica" Sokoll

utopiArte

Als Antwort auf Rainer "friendica" Sokoll • 3 Tage her • •

@Rainer "friendica" Sokoll

I am confused. If you do not have ssh access, sftp will fail.

two server

server 1
shared hosting
hosts the file
access over sftp
no access over SSH
access over https, locating the file in a public folder, access with wget, works at least up to 1.9gig
server2
VPS hosting
access over SSH
never intended or tested to access over sFTP till now
needs/requests the 3.7gig file

Probable sFTP option:
rootName@VPShosting:# someSFTPtool sftpUSERNAME@sftpsSharedhostingID/sharedHostingstoragefolder/file.tar.gz /thisVPSserver/localstorage/file.tar.gz

@Rainer "friendica" Sokoll

teilten dies erneut

⇧