utils¶

This module provides funtions that are not DataFrame-centric, but are helpful to have whilst using Spark clusters interactively.

manatee.utils.silence(context, warn=False)[source]¶

Silences the PySpark log messages.

Removes INFO ( and optionally WARN ) messages from the specified context’s logger.

Parameters:

context : SparkContext, SQLContext, or HiveContext

For example, if sc is your SparkContext, calling silence(sc) will switch off INFO and WARN messages issued from this SparkContext.

warn : bool.

If True, the logger’s level is set to WARN rather than ERROR, so the context will continue to issue WARN messages as well as ERROR messages. If False, only ERROR messages are issued.