sparkpipelineframework.testing

v3.0.7

Testing Framework for SparkPipelineFramework For more information about how to use this package see README

Latest version published 28 days ago

License: Apache-2.0

PyPI

GitHub

Package Health Score

66 / 100

security
No known security issues
popularity
Limited
maintenance
Healthy
community
Limited

Popularity

Limited

GitHub Stars: 2
Forks: 0
Contributors: 10

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package sparkpipelineframework.testing, we found that it has been starred 2 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Oct 8, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

We found a way for you to contribute to the project! Looks like sparkpipelineframework.testing is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Healthy

Commit Frequency

Open Issues: 0
Open PR: 11
Last Release: 28 days ago
Last Commit: 2 months ago

Further analysis of the maintenance status of sparkpipelineframework.testing based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy.

We found that sparkpipelineframework.testing demonstrates a positive version release cadence with at least one new version released in the past 3 months.

In the past month we didn't find any pull request activity or change in issues status has been detected for the GitHub repository.

Community

Limited

Readme: Yes
Contributing.md: Yes
Code of Conduct: No
Contributors: 10
Funding: No

This project has seen only 10 or less contributors.

We found a way for you to contribute to the project! Looks like sparkpipelineframework.testing is missing a Code of Conduct.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: >=3.10

Age: 4 years
Latest Release: 28 days ago
Dependencies: 0 Direct
Versions: 136
Maintainers: 1
Wheels: OS Independent

Readme

SparkPipelineFramework.Tests

Testing framework that can tests SparkPipelineFramework library by just providing input files to setup before running the transformer and output files to use for verifying the output

Usage

Create a folder structure similar to the folder structure of your library in SparkPipelineFramework (This is how the Testing Framework finds the Transformer to run)
Create an input folder and put in files that represent the input views. These files can be csv, json or parquet
(Optionally) Create an input_schema folder and put in any schemas you want applied to the above views. This follows the Spark Json Schema format.
(Optional) Create an output folder and put in files that represent the output views you expect. These files can be csv, json or parquet
(Optional) Create an output_schema folder and put in any schemas you want applied to the output views
Copy the following test code and put it in a test file in this folder

from pathlib import Path

from pyspark.sql import SparkSession

from spark_pipeline_framework_testing.test_runner import SparkPipelineFrameworkTestRunner


def test_folder(spark_session: SparkSession) -> None:
    data_dir: Path = Path(__file__).parent.joinpath('./')

    SparkPipelineFrameworkTestRunner.run_tests(spark_session=spark_session, folder_path=data_dir)

Now just run this test.

Note: the test finds files in sub-folders too.

Example

For the transformer defined here: https://github.com/imranq2/SparkPipelineFramework.Testing/tree/main/library/features/people/my_people_feature You can find the test here: https://github.com/imranq2/SparkPipelineFramework.Testing/tree/main/tests/library/features/people/my_people_feature

Publishing a new package

Create a new release
The GitHub Action should automatically kick in and publish the package
You can see the status in the Actions tab

FAQs

What is sparkpipelineframework.testing?

Testing Framework for SparkPipelineFramework. Visit Snyk Advisor to see a full health score report for sparkpipelineframework.testing, including popularity, security, maintenance & community analysis.

Is sparkpipelineframework.testing popular?

The python package sparkpipelineframework.testing receives a total of 932 weekly downloads. As such, sparkpipelineframework.testing popularity was classified as limited. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is sparkpipelineframework.testing well maintained?

We found that sparkpipelineframework.testing demonstrated a healthy version release cadence and project activity. It has a community of 10 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is sparkpipelineframework.testing safe to use?

The python package sparkpipelineframework.testing was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 31 October-2024, at 22:50 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP