A financial services company needs to ingest daily transaction data from an external vendor's SFTP server into a Lakehouse in Microsoft Fabric. The SFTP server requires authentication using an SSH key pair for security. The data engineering team plans to use a Data Pipeline for this ingestion. Which sequence of steps is required to securely configure the connection to the SFTP server?
- AInstall the SSH private key on an on-premises data gateway, then create a linked service for SFTP that uses the gateway.
- BCreate a linked service for SFTP, specify hostname and port, then upload the SSH private key file directly into the linked service definition.
- CCreate an Azure Key Vault secret for the SSH private key content, then create a linked service for SFTP referencing the Key Vault secret.
- DUse a Managed Identity for the Data Pipeline to authenticate directly with the SFTP server, after assigning appropriate roles.
Show answer & explanationAnswer & explanation
Correct answer: C. Create an Azure Key Vault secret for the SSH private key content, then create a linked service for SFTP referencing the Key Vault secret.
For secure handling of sensitive credentials like SSH private keys in Microsoft Fabric Data Pipelines, it is best practice to store them in a secure secret store like Azure Key Vault. The Data Pipeline's SFTP linked service can then be configured to retrieve the private key content from Key Vault at runtime, ensuring the key is never directly exposed or hardcoded in the pipeline definition.
Why the other options are wrong
- A. An on-premises data gateway is used for connecting to on-premises data sources, not for securely storing SSH keys for an external SFTP server accessible over the internet.
- B. Uploading the private key directly is a security risk as it exposes the key within the service definition, which is not recommended for production.
- D. SFTP servers typically do not support Azure Managed Identity for authentication; they require username/password or SSH key pairs. Managed Identity is for authenticating to Azure services.
Secure Credential Management (SFTP)
The practice of securely storing and accessing sensitive authentication information, such as SSH private keys, for external SFTP connections in Microsoft Fabric Data Pipelines.
- Azure Key Vault is the recommended secure store.
- Linked services reference Key Vault secrets.
- Prevents hardcoding or exposing credentials.
Memory trick: Key Vault is the vault for your SSH key, unlocking secure data transfer.