php - Could I fill and filter this array more efficiently?

I have a reasonably sized multi-dimensional associative array that I fill then filter out null values with a recursive array filter to produce JSON that gets sent to an API. The problem is that when I feed 7,000+ products into it, PHP runs over memory limits and time limits. I've increased the runtime and memory limit to get around this, but if anyone can suggest how I might shave even small amounts of time off each product, it will make a significant difference.

I have thought about perhaps using array_fill_keys() to somehow not populate the value in the array if it's empty, null or FALSE and that would get rid of the need for the array_filter_recursive() function, but it seems to me that this would make the code much less readable (than it already is) and perhaps not save any compute time.

As is now, the script takes about 2.5-3 minutes to run 7k products.

Here is the main function that takes in $data which is an array of 1 or more products. Using hrtime() to measure the foreach($data as $product) loop, It's taking ‭0.018452269 seconds for one product, and 131.546232569 seconds for 7129 products.

    private function populate_api_data($data)
    {
        $config_data    = array();
        $api_data       = array();

    //Populate Config data
        foreach($this->CI->Appconfig->get_all()->result() as $app_config)
        {
            $config_data[$app_config->key] = $app_config->value;
        }

        foreach($data as $product)
        {
            $item_id            = $product['item_id'];
            $number_of_discs    = $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_numberofdiscs'])->attribute_decimal;
            $number_of_pages    = $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_numberofpages'])->attribute_decimal;
            $running_time       = $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_runningtime'])->attribute_decimal;
            $stock_on_order     = $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_stockonorder'])->attribute_decimal;

            $api_data[]         = array(
                'AspectRatio'           => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_aspectratio'])->attribute_value,
                'AudienceRating'        => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_audiencerating'])->attribute_value,
                'AudioFormat'           => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_audioformat'])->attribute_value,
                'AudioTrackListing'     => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_audiotracklisting'])->attribute_value,
                'AuthorsText'           => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_authorstext'])->attribute_value,
                'Barcode'               => $product['item_number'],
                'Binding'               => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_binding'])->attribute_value,
                'BookForeword'          => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_bookforeword'])->attribute_value,
                'BookIndex'             => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_bookindex'])->attribute_value,
                'BookSampleChapter'     => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_booksamplechapter'])->attribute_value,
                'Contributors'          => $this->get_contributor_ao_array($item_id),
                'Condition'             => $this->get_condition_ao_array($item_id, $config_data['clcdesq_condition']),
                'DateAdded'             => $this->get_date_added($item_id),
                'Depth'                 => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_depth'])->attribute_decimal,
                'Description'           => $product['description'],
                'DimensionUnit'         => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_depth'])->attribute_decimal !== NULL ? $this->CI->Attribute->get_info($config_data['clcdesq_depth'])->definition_unit : NULL,
                'DiscountGroup'         => $this->get_product_discount_group_ao_array($item_id),
                'EAN'                   => $this->get_ean($this->get_isbn($product['item_number'])),
                'Format'                => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_format'])->attribute_value,
                'Height'                => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_height'])->attribute_decimal,
                'InternalCode'          => (string)$item_id,
                'ISBN'                  => $this->get_isbn($product['item_number']),
                'KindId'                => $product['category'] == 'Books' ? 1 : NULL,      /* Regular Book*/
                'Language'              => $this->get_language_ao_array((int)$item_id),
                'MediaType'             => $this->get_media_type_ao_array($product['category']),
                'NumberOfDiscs'         => $number_of_discs ? (int)$number_of_discs : NULL,
                'NumberOfPages'         => $number_of_pages ? (int)$number_of_pages : NULL,
                'OriginalTitle'         => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_originaltitle'])->attribute_value,
                'Price'                 => (float)$product['unit_price'],
                'PriceWithoutVAT'       => (float)$this->get_price_without_VAT($product['unit_price']),
                'PriceNote'             => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_pricenote'])->attribute_value,
                'Producer'              => $this->get_producer_user_ao_array($item_id),
                'ProductStatusProducer' => $this->get_product_status_producer_ao_array($item_id),
                'PriceCurrency'         => $config_data['currency_code'] !== '' ? $config_data['currency_code'] : NULL,
                'Published'             => $product['deleted'] == FALSE ? TRUE : FALSE,
                'PublisherRRP'          => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_publisherrrp'])->attribute_decimal,
                'ReducedPrice'          => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_reducedprice'])->attribute_decimal,
                'ReducedPriceStartDate' => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_reducedpricestartdate'])->attribute_date,
                'ReducedPriceEndDate'   => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_reducedpriceenddate'])->attribute_date,
                'ReleaseDate'           => $this->get_release_date($item_id,$config_data['clcdesq_releasedate']),
                'RunningTime'           => $running_time ? (int)$running_time : NULL,
                'Series'                => $this->get_product_series_ao_array($item_id),
                'StockCount'            => empty($product['stock_count']) ? (int)$this->get_total_quantity($item_id) : $product['stock_count'],
                'StockOnOrder'          => $stock_on_order ? (int)$stock_on_order : NULL,
                'Supplier'              => $this->get_supplier_user_ao_array($product['supplier_id']),
                'ShowOnWebsite'         => empty($product['show_on_website'])? $this->get_show_on_website($item_id, $config_data['clcdesq_showonwebsite']) : $product['show_on_website'],
                'Subtitle'              => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_subtitle'])->attribute_value,
                'Subtitles'             => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_subtitles'])->attribute_value,
                'TeaserDescription'     => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_teaserdescription'])->attribute_value,
                'Title'                 => $product['name'],
                'UniqueId'              => $this->get_uniqueid($item_id, $config_data['clcdesq_uniqueid']),
                'UPC'                   => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_upc'])->attribute_value,
                'VatPercent'            => (float)$this->CI->Item_taxes->get_info($item_id)[0]['percent'],
                'VideoTrailerEmbedCode' => $product['videotrailerembedcode'],
                'Weight'                => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_weight'])->attribute_decimal,
                'WeightForShipping'     => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_weightforshipping'])->attribute_decimal,
                'WeightUnit'            => $this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_weight'])->attribute_decimal !== NULL ? $this->CI->Attribute->get_info($config_data['clcdesq_weight'])->definition_unit : NULL,
                'Width'                 => (float)$this->CI->Attribute->get_attribute_value($item_id, $config_data['clcdesq_width'])->attribute_decimal,
                'Categories'            => array($this->get_category_ao_array($item_id, $this->CI->Item->get_info($item_id)->category, 0))
            );
        }

        $api_data = array('Products' => $this->array_filter_recursive($api_data));

        return $api_data;
    }
Copy code

each get_attribute_value function call makes an SQL query so it's possible with 30+ MySQL calls each over 7,000+ items that the bulk of the time is being spent on those.

Here is the array_filter_recursive function. Using hrtime() to measure, It's only taking 0.080871ms for one product, and 0.783339657 seconds for 7129 products So optimization here won't make a huge impact.

    private function array_filter_recursive($input)
    {
        foreach($input as $key => &$value)
        {
            if(is_array($value))
            {
                $value = $this->array_filter_recursive($value);
            }

            if(in_array($value,array(NULL,'')) && $value !== 0)
            {
                unset($input[$key]);
            }
        }

        return $input;
    }
Copy code

populate_api_data has a number of private function calls in it and I recognize that each of those needs to be optimized as well. I can post those too, but I was also hoping that there is something I'm missing in the main logic that could be optimized significantly.

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.