PANDAS I

Owned by ARGOSLABS
Dec 01, 2023
1 min read



PANDAS I

Author: Jerry Chae

(warning) ARGOS LABS will be releasing more pandas plugins in the future (pandas II, III, IV, V, and VI)


This plugin brings some of the pandas solutions to your bot and automation projects.

pandas is one of the most popularly used data manipulation software by data scientists and researchers around the world.  Additional information about pandas are available from the links below,

prerequisite

Anyone can use this tool, but it is recommended to have basic Python and Regular Expression skills.




Need help?

Technical contact to tech@argos-labs.com


May you search all operations,


How to set Advanced parameters.

Please also refer to:




  1. Specify file-path of input and output files. If you want give Title to the HTML report, put the Title here. If you wan to analyze a certain range from the data file specify here.

  2. When reading Excel file, specify sheet name.

    • Defaults to 0: 1st sheet as a DataFrame
    • 1: 2nd sheet as a DataFrame
    • "Sheet1": Load sheet with name “Sheet1”
    • [0, 1, "Sheet5"]: Load first, second and sheet named “Sheet5” as a dict of DataFrame
    • None: All sheets.

3. Specify what row you have the headers: Row (0-indexed) to use for the column labels of the parsed DataFrame. If a list of integers is passed those row positions will be combined into a MultiIndex. Use None if there is no header.

4. Column (0-indexed) to use as the row labels of the DataFrame. Pass None if there is no such column. If a list is passed, those columns will be combined into a MultiIndex. If a subset of data is selected with usecols, index_col is based on the subset.

5. If None, then parse all columns.

    • If str, then indicates comma separated list of Excel column letters and column ranges (e.g. “A:E” or “A,C,E:F”). Ranges are inclusive of both sides.
    • If list of int, then indicates list of column numbers to be parsed.
    • If list of string, then indicates list of column names to be parsed.

6. Data type for data or columns. E.g. {‘a’: np.float64, ‘b’: np.int32} Use object to preserve data as stored in Excel and not interpret dtype. If converters are specified, they will be applied INSTEAD of dtype conversion.

7. Delimiter to use. If sep is None, the C engine cannot automatically detect the separator, but the Python parsing engine can, meaning the latter will be used and automatically detect the separator by Python’s builtin sniffer tool, csv.Sniffer. In addition, separators longer than 1 character and different from '\s+' will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.

8. Return Value stores the complete file-path of “Out File”.


Functions supported. (Execution sequence must be in the order listed below.)

  1. Extraction of rows that meet logical criteria (Filtering).
  2. Replacing of columns.
  3. Adding new columns.
  4. Dropping (removing) of columns.
  5. Selection of rows and columns by position.


Command Examples

1)  Extraction of rows that meet logical criteria (Filtering).

 

For Column "Qty" filter out less then 500

df['Qty'] >= 500.0
For Column "Item ID" filter only starts with “The”. Function “contains” take the parameter a regular expression.
df['Item ID'].str.contains('^The')



regex (Regular Expressions) Examples
'\.'Matches strings containing a period '.'
'Length$'Matches strings ending with word 'Length'
'^Sepal'Matches strings beginning with the word 'Sepal'
'^x[1-5]$'Matches strings beginning with 'x' and ending with 1,2,3,4,5
'^(?!Species$).*'Matches strings except the string 'Species'



For logical AND filter
(df['Qty'] >= 500.0) & (df['Item ID'].str.contains('^The'))



Logic in Python (and pandas)
<Less than!=Not equal to
>Greater thandf.column.isin(values)Group membership
==Equalspd.isnull(obj)Is not NaN
<=Less than or equalspd.notnull(obj)Is not NaN
>=Greater than or equals$, |, ~, ^, df.any(), de.all()Logical and, or, not, xor, any, all


2) Replacing of columns.


Replace "A " with "B" for column "Col"
Col ::= 'A','B'


3) Adding new columns.


Extract first word from “Col” column and then add it to the new column “NewCol”
NewCol=lambda x: x['Col'].str.extract(r'^(\w+)')


4) Dropping (removing) of columns.

 

Delete one or more columns
‘Length’, ’Hight’


5) Selection of rows and columns by position.

Select between 3rd and 5th rows and 1st and 3rd columns
2:5, 0:3



NOTE

Execution sequence of the functions must be in the order listed above within “one run” of the pandas I plugin. If you require to execute functions out of this order, please use multiple pandas I plugin in back to back. (In a daisy chain way)

How to set parameters – actual examples.


About Jupyter Notebook --- Before building your automation with the pandas I plugin, we strongly recommend you to take your data and test your commands on Jupyter Notebook https://jupyter.org/. You can use the commands from Jupyter Notebook directly at the pandas I plugin.




All Plugins