Executing a Task using a Cloud Engine

Task outline

In this tutorial, you first connect to Talend Cloud Portal. To do this, you must have an administrator account, or you can create a trial account at https://iam.us.cloud.talend.com/idp/trial-registration (for which you need an email address).

Then, you publish a predefined Job and run it on Talend Cloud.

You also see how to execute a task using a Cloud Engine.

A simple, predefined Job is on your training machine. This Job reads a customer’s file from Dropbox, applies a simple transformation to it (concatenates the first and last name), and writes a results file to Dropbox.

Configuring your virtual machine

  1. To start the virtual machine, open a new tab with the same page and click the following link : START VM!

    The VM is launched in your web browser. Wait for Windows to start.

    A script is automatically launched. You can close it by clicking the X button.

  2. Allow the PC to be discoverable on the network.

    On the Networks panel to the right, click the Yes button.

    This is critical in order to correctly execute all the exercises in this tutorial.

Connecting to the Talend Cloud Portal

  1. Log in the Talend Cloud Portal URL.

    1. Go to https://portal.us.cloud.talend.com and click CONTINUE.

    2. Log in to the Talend Cloud Portal interface. Notice that you can use Asia-Pacific, European or US Instance. If, for regulatory or other reasons, you must have your data hosted in Europe, you can change the instance from US to European.

  2. Connect to the Talend Cloud portal using your administrator account.

    If you cannot locate your log-in credentials, sign up for a trial at https://iam.us.cloud.talend.com/idp/trial-registration.


    Enter your email and click START YOUR TRIAL. You immediately receive a Talend Cloud – Trial Registration email with a link to complete registration. Follow the link and enter your information and Password and click COMPLETE REGISTRATION.

    You finally receive a Talend Cloud Account Confirmation email with your log in credentials. More precisely, look for a user name ending in @domain.talend.com or @trialxxxxxx.talend.com.

    To log in to the graphical interface, you can either use the email or the complete user name. Even if you use the email to log in, note the complete user name, as you need it later in this course.

    In the screenshots for this course, Adam Brown is the administrator.

    His user name is abrown@training.talend.com

    Click LOGIN.

  3. The portal welcome page opens.

    Notice that several applications are available on the screen: Management Console, Studio, Data Preparation, and Data Stewardship.

    Clicking on SELECT AN APP at the top left of the page, you can also open applications by selecting them from the drop-down menu.

    In this tutorial, you use Management Console to configure and execute your Talend Studio Jobs in the cloud.

Retrieving the complete username from Talend Management Console

  1. Click the Management Console section on the welcome page.

    The Talend Management Console welcome page opens. Click the LAUNCH button to access the tab in your browser.

  2. To display the user details, on the USERS tab on the left menu pane, click your username.

  3. Copy the complete Login name, which you need in order to configure Studio.

  4. Click the CANCEL button.

Connecting Talend Studio to Talend Cloud

You are ready to configure Talend Studio so it can publish Jobs in the cloud.

  1. Double-click the Talend Studio logo to open Talend Studio.

  2. Select Local_Project and click Finish.

  3. When the project opens, go to Window > Preferences > Talend > Integration Cloud.

  4. Enter your Talend Cloud Portal account credentials.

    Replace the default value in the Account Username text box with the login nameyou copied from Talend Management Console. Look for a username ending in @training.talend.com or @trialxxxxxx.talend.com.

    Enter your password (the same one you used to access the Talend Cloud portal).

  5. Ensure that access is granted by clicking the Test Connection button. “Service available” appears in green.

  6. Click Apply and OK.

    Talend Studio is configured.

  7. Optional (if your connection is slow) : go to Talend > Performance and set the Connection Timeout with Administrator Center field parameter to 300 seconds.

When your Tasks run on Talend Management Console application, you use a cloud engine if all data or applications are in the cloud.

You can also leverage remote engines and clusters when on-premises applications and data are involved.

Creating a Dropbox account

If you do not have a Dropbox account, complete the following steps to sign up for a trial account.

  1. Create a Dropbox account.

    1. Open a browser and connect to https://www.dropbox.com.
    2. Sign up for a new account: fill in your first name, last name, email address, and password; accept the conditions, and click Sign up.

      Your account is created; you do not need to download Dropbox.

    3. Click the back button to connect to Dropbox.
    4. Click the Upload, then Files button.

    5. Go to C:/StudentFiles, select the customers.csv file, and click Open.

    6. Select Dropbox and click Upload.

    7. Click Files to verify that the file has been uploaded to Dropbox.

  2. Create a Dropbox application.
    1. Open a new tab and connect to https://www.dropbox.com/developers/apps.

    2. If you are not already logged in, sign in using your credentials.

    3. Click the Create app button.

    4. If the Dropbox website asks for an email verification, click Send email for verification.

    5. Click on the Verify your email button on the email you received.

    6. Refresh the Dropbox application page to finish application creation.

    7. Select Dropbox API and Full Dropbox access.

      Name your app TIC_<yourlogin>.

      Agree the terms and conditions and click the Create app button.

    8. The Dropbox application has been created.
  3. Generate the Dropbox token.

    1. Click the Generate button.

    2. Copy the Generated access token, which you need in this exercise.

Updating a Job in Talend Studio

  1. In the Repository, expand Job Designs> Standard and open the customers_dropbox Job.

  2. Click the Contexts tab to open it.

  3. Click the Value field to update the context variable value with your Dropbox token (keep the quotation marks).
  4. Click Run to execute the Job.

    The Job downloads the customer.csv file from Dropbox, concatenates the first name and last name for all customers in the file and writes the results to a customers_results.csv file in your Dropbox account.

  5. Go back to your Dropbox account and verify that the file has been uploaded.

    Open the file and confirm that the customer name is now a concatenation of first name and last name.

  6. Delete the customers_results.csv file from Dropbox.

In the next section, you will deploy your Job to the cloud and you will execute it there.

Publishing a Job to Talend Management Console

Once Talend Studio is configured with your Talend Cloud account, you can publish a Job to the cloud.

  1. From the Talend Studio repository, right-click on the Job’s name, then select Publish to Cloud.

    If the Publish to Cloud pop-up window disappears, click on the Designer space so it appears again.

  2. Leave the default workspace and click Finish.

  3. When the Job upload is complete, click OK.
  4. Connect to the Talend Cloud portal, and open the Talend Management Cloud application.

    Open MANAGEMENT tab, and click Tasks icon in your Personal space.

  5. Verify that the Job you published has been encapsulated in a task by the same name.

Creating a Dropbox connection in Management Console

To execute your Job in the cloud, you also need to configure the Dropbox connection in TMC.

  1. In Management Console, on the menu on the left, click Management.

  2. Select Connections in your Personal space.



  4. Configure the Add New Connection section.

    1. Leave Workspace as Personal (yourName)

    2. For Application, select Dropbox

    3. Define connection Name as TIC_Dropbox and paste in the generated access token.

  5. Click Save and confirm that the connection has been created.

Configuring and running a task using a cloud engine

When the Talend Studio Job has been published to Talend Cloud and encapsulated within a task, and a connection to Dropbox has been created, the task can be executed in the cloud.

  1. In the Management Console application, on the menu on the left, select MANAGEMENT tab, and click Tasks icon in your Personal space.

  2. The Tasks and Plan list appears. To access details, click your Task name.

  3. To define the Task settings, hover over the Configuration section until the pencil symbol appears, then click it to edit configuration.

  4. Leave the Artifact configuration as defined, and click CONTINUE.

  5. In the Connection parameters, select the Dropbox connection you created earlier. Click CONTINUE.

  6. Configure the Go Live section.
    1. Leave the Runtime as Cloud.

    2. To execute the task immediately, keep the Run type field as Manually.

    3. Click the GO LIVE button.

      A pop-up window confirms that the task execution has started.

      Back on your task screen, you can see in the RUN HISTORY section that the task is running.

  7. Control the task execution.
    1. In the RUN HISTORY section, check the execution status by clicking the REFRESH button.

    2. When the task execution is complete, the duration is displayed.

    3. To access detailed information on the task execution, expand panel by clicking the arrow on the right, and click VIEW LOGS. From here, you can filter by user logs or developer logs, or display all logs, including information messages.

  8. Go back to Dropbox and verify that the customers_results.csv file has been added.

Scheduling task execution

If you don’t want to execute the task immediately, you can schedule the task execution.

  1. From your Task and Plan list in TMC, go back to the customers_dropbox details in the Configuration section to edit your task.

  2. Configure the Go Live section.

    1. Leave Runtime as Cloud.

    2. For Run type, select Daily.

    3. Define the reference time zone.

    4. Set Starts and Repeat from to the current date and time, and set Repeat to 10 minutes after.

    5. For Repeat every, select 1 day(s), At specific intervals and Repeat every 5 minutes.

  3. Click the GO LIVE button.

    A pop-up window confirms that the task execution has been scheduled.

    On the task screen, you can see in the RUN HISTORY section that the task is running.

  4. Go back to Dropbox and verify that the file has been added.

Talend Cloud also allows you to create an execution plan. By using this operation, you can execute several tasks in sequence and schedule your execution plan.

For further details, see other Talend resources:

You have finished the tutorial.