Portugal
ambi.
Associação Porto Digital
Data Engineer
Portugal
Porto, Portugal
PythonSQLDocker
Automatic translation. Check the original.
About the opportunity
Associação Porto Digital, a private non-profit organization based in Porto, with over 20 years of existence and growing, intends to hire a Data Engineer.
The desired professional will report to the Coordinator of the Data Architecture and Artificial Intelligence area and will join a hybrid work team (on-site and remote work). The employment contract will initially be fixed-term with the possibility of renewal and/or indefinite term depending on the demonstrated experience.
Join Associação Porto Digital as a Data Engineer and drive urban digital transformation in Porto. Improve public services, impact citizens' lives, and contribute to sustainable development.
TASKS
Main activities/tasks to perform:
- Design, develop, and maintain data pipelines (batch and streaming) for structured and unstructured data.
- Develop and optimize ETL processes using programming languages such as Python and/or SQL.
- Work with open-source technologies for data processing, integration, and storage (e.g., Spark, Kafka, NiFi, Airflow) in an on-premises environment.
- Manage and optimize databases, ensuring performance, integrity, and availability.
- Ensure data quality, integrity, and reliability throughout the entire lifecycle (validation, monitoring, and testing).
- Design and improve data architectures, ensuring performance, scalability, and security.
- Establish the connection between Data Scientists, Data Analysts, and Artificial Intelligence by transcribing needs into technical requirements and data processing;
- Improve the city of Porto's Urban Platform, creating new ways to increase the value and quality of data collected and used in decision-making with an impact on the community's quality of life.
REQUIREMENTS
Education
- Degree in Computer Engineering, Data Analysis, Computer Science, Electrical and Computer Engineering, or similar.
Qualifications
- Minimum of 3 years of experience in Data Engineering roles, data platform development, or similar areas.
- Good knowledge of SQL and experience with relational and analytical databases.
- Experience in developing ETL/ELT pipelines for data integration, transformation, and provisioning.
- Good knowledge of Python for data processing, automation, and system integration.
- Hands-on experience with Apache Kafka, including understanding topics, partitions, keys, consumer groups, offsets, retention, and replay mechanisms.
- Understanding of the main challenges in real-time data processing, namely ordering, duplication, idempotency, latency, error handling, and schema evolution.
- Experience with pipeline orchestration tools, such as Apache Airflow or equivalents.
- Knowledge of data modeling, data warehouses, data lakes, and lakehouse architectures.
- Experience in implementing incremental processes, quality control, and pipeline monitoring.
- Familiarity with Git, code review processes, and continuous integration and delivery practices.
- Knowledge of Docker and principles of developing and operating services in distributed environments.
- Good knowledge of English.
Personal and organizational skills
- Autonomy, proactivity, and sense of responsibility.
- Good analytical and structured problem-solving skills.
- Ability to design simple, reusable, and sustainable solutions, avoiding isolated developments per use case.
- Organization, rigor, and attention to detail.
- Focus on quality, reliability, and continuous improvement.
- Ability to diagnose problems in pipelines and distributed systems.
- Ability to work in multidisciplinary teams and collaborate with technical and functional profiles.
- Good communication and ability to clearly explain technical decisions.
- Ability to document processes and promote knowledge sharing.
- Results-oriented and deadline-driven.
- Interest in working with different urban domains, such as mobility, economy, environment, territory, and municipal services.
- Focus on creating value for the city of Porto and its citizens.
Nice to have
- Experience with ClickHouse or other columnar databases and with dbt.
- Knowledge of Apache NiFi, Change Data Capture with Debezium, or stream processing with Flink/Kafka Streams.
- Familiarity with lakehouse architectures, Apache Iceberg, Parquet, and object storage.
- Experience with geographic data, PostGIS, or GIS systems.
- Knowledge of FIWARE, NGSI-LD, Orion-LD, data catalogs, or data lineage mechanisms.
BENEFITS
- Flexible hours
- Meal allowance
- Health insurance
- Mobile phone and communication service
- ANDA APP
More opportunities with one profile
Want to be considered for more roles without repeating the whole process? Create your profile and be found by companies using Nort.
Create my profile
