I have an ETL process that currently brings data from a remote DB into HDFS and then uses bcp from the mssql-tools package to transfer data as TSVs from the linux machines into a Windows MSSQL DB. I would like to set up this process using Airflow on another linux server. To do this, I plan to mount the HDFS folder via NFS onto the Windows machine and run an SSIS package to transfer the data into the MSSQL DB. Is there a way/operator to run commands on a remote Windows machine from Airflow? Eg.
"C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\DTExec.exe" /FILE "c:\users\me\documents\visual studio 2012\Projects\etl_flatfiles\Integration Services Project1\Package.dtsx" /DECRYPT "password"
Related
I'm trying to create a local development environment to test website changes out in. The web developers I'm working with have a gitlab repo of the website with auto DevOps where changes are pushed to the production/stage sever when commited.
I downloaded Laragon and was able to serve pages locally, then I cloned the gitlab repo into my root directory.
How do I get a copy of production website's database and then connect it to my local development environment with laragon? I tried to duplicate the database on production with WP all-in-one migration but there are read/write restrictions.
I was advised the following: "I would recommend copying the database onto your local MariaDB/MySQL server the Laragon provides. You'd need to get a dump of the database from the server using SSH command line tools and then you can use the phpMyAdmin interface provided by Laragon to upload the dump".
I wasn't able to SSH from the windows command line, but I could SSH into the server using Putty, however, how am I able to transfer the file to my local windows machine? If I'm logged into my server wouldn't the file just be placed somewhere in the file system of the remote machine, so I wouldn't be able to use it with Laragon.
If you are using Laragon Full, you have ssh/scp command in Laragon's Terminal. So you can get your-sql-dump from your remote server easily.
Here's how:
Open Laragon's Terminal:
Menu > Laragon > Terminal
Run this command:
scp user#your-remote-host:/path-to-your-sql-dump C:/laragon/tmp/your-sql-dump
Import your sql-dump to your local database:
mysql -u root -p your-database < C:\laragon\tmp\your-sql-dump
Reference: https://laragon.org/download/migrate-from-xampp.html
That's all.
I have VS2015 with RTools and RTVS on a windows 10 machine.
I also have a Server 2012 with SQL Server, with R Server, and RTVS R Services installed.
I cannot connect the Win-10 RTVS to the RServer using Remote connections. I have done the following things:
On the Server:
Installed the Certificate
Installed R Services. Found both Services R Host Broker Service and R User Profile Service alive and running. The associated json is also set.
Found both programs listed on the Windows Firewall for allowed on inbound and outbound connections
Ensured .net 4.6.1 is set
On the Client Workstation:
Installed R Tools for Visual Studio on top of Visual Studio 15 (SSDT)
Installed Microsoft R and other related programs.
Copied and added the certificate from the server to the workstation's list of trusted certificates.
Ensured .net 4.6.1 is set
Added Server to Workstation's list of remote workspaces in VS. Called the workspace 'simulator'
I can telnet to rserver.domain.com with port 5444 to ensure port is listening.
however, when I go to connect to the remote workspace I get this error:
Connecting to R Workspace failed.
Reason: Machine 'Simulator' appears to be online, but the Remote R Service is not running.
>
I'm missing something, and don't know what. Where do I go from here?
Thanks.
I am performing some automate validation in Unix machine using jenkins. I have to trasnfer a file from Unix machine to my local windows machine to upload it in HP ALM.
Is there any automation script which ca perform this task?
Launch your jenkins slave via Java web start (Manage Jenkins > Manage Nodes > Launch method), you need to specify remote root directory where you need
once your slave is configured go to windows machine and inside your browser hit jenkins server
and now start your slave which will open a pop up window and everything goes well it will show as connected
Now inside your job> build settings restrict your build to be run on windows slave you have configured.
You are set and your files will get copied from unix machine to windows wuth use of jenkins
i am creating a web application in asp.net with database on postgres. i have successfuly created a database backup by putting the destination of database backup on the webconfig file. the web application succesfully creates backups when published on windows server, however i was wondering if the server is changed to linux the destination folder would be invalid.what could be the solution for it to work on different OS??
another problem is that the postgres installation could be on any drive in linux whereas rightnow in my webapplication i open the system diagnostics to run the pgdump command using the info.filename="C:\Program Files\PostgreSQL\9.3\bin\" + commandType + ".exe ";
anyone has solution for multiOS database backup for postgres using asp.net?? is there any solution to this??
You can continue to use your backup scripts on Windows. pg_dump will happily connect to the remote database on the Linux box. E.g.
pg_dump -h my_linux_server ....
I strongly suggest reading the PostgreSQL docs on backup/restore. Consider setting up PITR in addition to periodic dumps. I'd recommend running PgBarman on the Linux server (archiving to an external drive, NAS, or network share) for point-in-time recovery, plus periodic dumps to the Linux box.
See: http://www.postgresql.org/docs/current/static/backup.html
I have some shell scripts on a Windows 2003 server. These scripts process some flat files and upload data to Oracle using sqlldr. Currently I'm using telnet to go into the box and run each of these scripts manually. Is there a way that these scripts can be run from a ASP.Net/Winforms application?
I'm doing similar things with Windows Communication Foundation. http://msdn.microsoft.com/en-us/library/ms731082.aspx