- Install, configure, and maintain Databricks clusters and workspaces.
- Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance.
- Implement and manage access controls and security policies to protect sensitive data.
- Perform regular software updates and patches.
- Optimize cluster configurations for performance and cost-effectiveness.
- Implement and enforce security policies, access controls, and encryption mechanisms.
- Stay up-to-date with security best practices and compliance requirements.
- Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
- Collaborate with data engineers to integrate Databricks with other data sources, data warehouses, and data lakes.
- Maintain detailed documentation of configurations, procedures, and best practices.
- Collaborate with cross-functional teams, including data scientists, data engineers, and business analysts to understand their requirements and provide technical solutions
Requirements
- Bachelor's degree in a relevant field (computer science, data engineering, etc.)
- 2 to 3 years experience with Databricks Platform
- Strong knowledge of Apache Spark and Databricks architecture.
- Familiar with managing clusters, workspaces, notebooks, user group management.
- Familiar with configuring permissions.
- Can create notebooks related to cleanup/housekeeping and schedule/manage jobs through Databricks
- Willing to be assigned in Ortigas (fully onsite)